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Abstract —This paper deals with the market-bidding problem 
of a cluster of price-responsive consumers of electricity. We 
develop an inverse optimization scheme that, recast as a bilevel 
programming problem, uses price-consumption data to estimate 
the complex market bid that best captures the price-response 
of the cluster. The complex market bid is defined as a series of 
marginal utility functions plus some constraints on demand, such 
as maximum pick-up and drop-off rates. The proposed modeling 
approach also leverages information on exogenous factors that 
may influence the consumption behavior of the cluster, e.g., 
weather conditions and calendar effects. We test the proposed 
methodology for a particular application: forecasting the power 
consumption of a small aggregation of households that took part 
in the Olympic Peninsula project. Results show that the price- 
sensitive consumption of the cluster of flexible loads can be largely 
captured in the form of a complex market bid, so that this could 
be ultimately used for the cluster to participate in the wholesale 
electricity market. 

Index Terms —Smart grid, demand response, electricity mar¬ 
kets, inverse optimization, bilevel programming 

I. Introduction 

We consider the case of a cluster of flexible power con¬ 
sumers, where flexibility is understood as the possibility for 
each consumer in the cluster to change her consumption 
depending on the electricity price and on her personal prefer¬ 
ences. There are many examples of methods to schedule the 
consumption of individual price-responsive loads (see, e.g., 
Q-0). The portfolio of flexible consumers is managed by 
a retailer or aggregator, which bids in a wholesale electricity 
market on behalf of her customers. We consider the case where 
such a market accepts complex bids, consisting of a series of 
price-energy bidding curves, consumption limits, and maxi¬ 
mum pick-up and drop-off rates. In this paper, we present a 
data-driven methodology for determining the complex bid that 
best represents the reaction of the pool of flexible consumers 
to the market price. 

The contributions of this paper are fourfold. The first 
contribution corresponds to the methodology itself: we propose 
a novel approach to capture the price-response of a pool of 
flexible consumers in the form of a market bid using price- 
consumption data. In this work, the price is given as the result 
of a competitive market-clearing process, and we have access 
to it only from historical records. This is in contrast to some 
previous works, where the price is treated as a control variable 
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to be decided by the aggregator or retailer. In |4j, for example, 
the relationship between price and consumption is first mod¬ 
eled by a Finite Impulse Response (FIR) function as in |5| 
and peak load reduction is achieved by modifying the price. 
Similar considerations apply to the works of |6)-[9j, where a 
bilevel representation of the problem is used: the lower-level 
problem optimizes the household consumption based on the 
broadcast electricity price, which is determined by the upper- 
level problem to maximize the aggregator’s/retailer’s profit. 
Another series of studies concentrate on estimating price- 
energy bids for the participation of different types of flexible 
loads in the wholesale electricity markets, for example, time- 
shiftable loads (101, electric vehicles (ED and thermostatically- 
controlled loads fl2| . Contrary to these studies, our approach 
is data-driven and does not require any assumption about the 
nature of the price-responsive loads in the aggregation. 

The second contribution lays in the estimation procedure: 
we develop an inverse optimization framework that results in a 
bilevel optimization problem. From a methodological point of 
view, our approach builds on the inverse optimization scheme 
introduced in & but with several key differences. First, 
we let the measured solution be potentially non-optimal, or 
even non-feasible, for the targeted optimization problem as 
in G3 and G3- Moreover, we extend the concept of inverse 
optimization to a problem where the estimated parameters may 
depend on a set of features and are also allowed to be in the 
constraints, and not only in the objective function. 

Third, we study heuristic solution methods to reduce the 
computing times resulting from the consideration of large 
datasets of features for the model estimation. We do not solve 
the resulting bilevel programming problem to optimality but 
instead we obtain an approximate solution by penalizing the 
violation of complementarity constraints following a procedure 
inspired by the work of (16) , Finally, we test the proposed 
methodology using data from a real-world experiment that was 
conducted as a part of the Olympic Peninsula Project OH- 

11. Methodology 

In this section, we describe the methodology to determine 
the optimal market bid for a pool of price-responsive con¬ 
sumers. The estimation procedure is cast as a bilevel program¬ 
ming problem, where the upper level seeks to minimize a norm 
of the estimation error, in particular, the absolute difference 
between the measured consumption and the estimated one, 
while the lower level ensures that the estimated consumption 
is optimal, given the reconstructed bid parameters and the 
electricity price. 
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A. Lower-Level Problem: Price-response of the Pool of Con¬ 
sumers 

The lower-level problem models the price-response of the 
pool of consumers in the form of a market bid, whose 
parameters are determined by the upper-level problem. The bid 
is given by 6 t = {cib,t, r“, rf, P t , Pt}, which consists of the 
marginal utility corresponding to each bid block b, the maxi¬ 
mum load pick-up and drop-off rates (analogues to the ramp- 
up and -down limits of a power generating unit), the minimum 
power consumption, and the maximum power consumption, 
at time t £ T = {t : t = 1... T}, in that order. The utility 
is defined as the benefit that the pool of users obtains from 
consuming a certain amount of electricity. The marginal utility 
a b ,t at time t is formed by b £ B = {b : b = 1... B} blocks, 
where all blocks have equal size, spanning from the minimum 
to the maximum allowed consumption. In other words, the 
size of each block is ‘g-* . Furthermore, we assume that the 
marginal utility is monotonically decreasing as consumption 
increases, i.e., a b ,t > a b +\ t for all times t. Finally, the total 
consumption at time t is given by the sum of the minimum 
power demand plus the consumption linked to each bid block, 
namely, x\ ot = P t + EbeB x b,t- 

Typically, the parameters of the bid may change across 
the hours of the day, the days of the week, the month, the 
season, or any other indicator variables related to the time. 
Moreover, the bid can potentially depend on some external 
variables such as temperature, solar radiation, wind speed, 
etc. Indicator variables and external variables, often referred 
to as features, can be used to explain more accurately the 
parameters of the market bid that best represents the price- 
response of the pool of consumers. This approach is potentially 
useful in practical applications, as numerous sources of data 
can help better explain the consumers’ price-response. We 
consider the I external variables or features, named Z l t for 
i £ L = {i : i = 1,...,/}, to be affinely related to 
the parameters defining the market bid by a coefficient a,. 
This affine dependence can be enforced in the model by 
letting a b<t = a° b + Eiez a i Z i,u rf = r u0 + E 


p t - i + e %b 


bGB 


,4-1 - Pt - ^2 x b,t < rf feT-i(lc) 


bee 


Xb,t < 

x b ,t > 0 


Pt-Et 


B 


b£ B,t£T (Id) 
b£B,t£T. (le) 


Equations ( |Tb} and ( |Tcj ) impose a limit on the load pick-up and 
drop-off rates, respectively. The set of equations defines 
the size of each utility block to be equally distributed between 
the maximum and minimum power consumptions. Constraint 
< P~e] ) enforces the consumption pertaining to each utility block 
to be positive. Note that, by definition, the marginal utility is 
decreasing in a: t (ab,t > a b +i,t), so one can be sure that the 
first blocks will be filled first. We denote the dual variables 
associated with each set of primal constraints as A“, Xf , i() b t 
and t b, . 

—b,t 

Problem ([T]) is linear, hence it can be equivalently recast as 
the following set of KKT conditions p~8) , where (f2a|-(f2cj) are 
the stationary conditions and ([2d|-([2gt enforce complemen¬ 
tarity slackness: 

- K + ^2 + V’b.t = a b,i - Pi (2a) 


K-K+i-K + K+i-'P, 
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b £ B,t £ T (2f) 
b£B,t£T. (2g) 
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B. Upper-Level Problem: Market-Bid Estimation Via Inverse 
Optimization 


Eiei a i Zi,t, Pt — P + Eiei a i %i,t, and Given a time series of price-consumption pairs (p t ,: 


P t = P_ + YftiGi a pZi,t- The affine coefficients af,af, af, 
af and af-, and the intercepts a' 


0 r u0 ^dO 
6’ ' 




,P_,P enter the 
model of the pool of consumers (the lower-level problem) as 
parameters, together with the electricity price. 

The objective is to maximize consumers’ welfare, namely, 
the difference between the total utility and the total payment: 


Maximize > 

Xb,t ' 

tGT 



(la) 


the inverse problem consists in estimating the value of the 
parameters 8 t defining the objective function and the con¬ 
straints of the lower-level problem Q such that the optimal 
consumption Xt resulting from this problem is as close as 
possible to the measured consumption x r f neas in terms of a 
certain norm. The parameters of the lower-level problem 9 t 
form, in turn, the market bid that best represents the price- 
response of the pool. 

In mathematical terms, the inverse problem can be described 
as a minimization problem: 


where x b ,t is the consumption assigned to the utility block b 
during the time t, a b ,t is the marginal utility obtained by the 
consumer in block b and time t, and p t is the price of the 
electricity during time t. For notational purposes, let 71 1 = 
{t : t = 2,... ,T}. The problem is constrained by 


Minimize E m\P t + E] Xb 't ~ 

tGT bGB 


meas 

x t 


(3a) 


subject to 


Pt 
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Constraints © are the upper-level constraints, ensuring 
that the estimated marginal utility must be monotonically 
decreasing. Constraints (3c) correspond to the KKT conditions 
of the lower-level problem 0- 

Notice that the upper-level variables 9 t , which are parame¬ 
ters in the lower-level problem, are also implicitly constrained 
by the optimality conditions (2) of this problem, i.e., by the 
fact that Xi must be optimal for 0- This guarantees, for 
example, that the minimum power consumption be positive 
and equal to or smaller than the maximum power consumption 
( 0 < P_ t < Pt)- Furthermore, the maximum pick-up rate 
is naturally constrained to be equal to or greater than the 
negative maximum drop-off rate (~rf < r t "). Having said 
that, in practice, we need to ensure that these constraints are 
fulfilled for all possible realizations of the external variables 
and not only for the ones observed in the past. We achieved 
this by enforcing the robust counterparts of these constraints 
©• An example is provided in the appendix. 

Parameter wt represents the weight of the estimation error at 
time t, in the objective function. These weights have a threefold 
purpose. Firstly, if the inverse optimization problem is applied 
to estimate the bid for the day-ahead market, the weights could 
represent the cost of balancing power at time t. In such a 
case, consumption at hours with a higher balancing cost would 
weigh more and consequently, would be fit better than that 
occurring at hours with a lower balancing cost. Secondly, the 
weights can include a forgetting factor to give exponentially 
decaying weights to past observations. Finally, a zero weight 
can be given to missing or wrongly measured observations. 

The absolute value of the residuals can be linearized by 
adding two extra nonnegative variables, and by replacing the 
objective equation (3a) with the following linear objective 
function plus two more constraints, namely, (4b) and ( [4c] ): 


T 

Minimize E w t( e t + e i ) ( 4a ) 

,e t 


subject to 


£ + E 

beB 


x b ,t - xT eas = e+ - e t - 

t&r 

(4b) 

ef > e t > 0 

t € T 

(4c) 

&b,t ^ &6+1, t 

t&r 

(4d) 

0- 


(4e) 


In the optimum, and when wt > 0, (4b) and (4c) imply 
that ej 1 " = x t — x™ eas if x t > x™ eas , else ej~ = x™ eas — x t . 
By using this reformulation of the absolute value, the weights 
could also reflect whether the balancing costs are symmetric 
or skewed. In the latter case, there would be different weights 
for ef and ejT. 


III. Solution Method 

The estimation problem (4) is non-linear due to the com¬ 
plementarity constraints of the KKT conditions of the lower- 
level problem ([2]). There are several ways of dealing with these 
constraints, for example, by using a non-linear solver |[20), by 


recasting them in the form of disjunctive constraints © or by 
using SOS1 variables (22). In any case, problem (7) is NP-hard 
to solve and the computational time grows exponentially with 
the number of complementarity constraints. Our numerical 
experiments showed that, for realistic applications involving 
multiple time periods and/or numerous features, none of these 
solution methods were able to provide a good solution to 
problem 0 in a reasonable amount of time. To tackle this 
problem in an effective manner, we propose the following two- 
step solution strategy, inspired by ©: 

Step 1: Solve a linear relaxation of the mathematical 
program with equilibrium constraints (4) by penalizing viola¬ 
tions of the complementarity constraints. 

Step 2: Fix the parameters defining the constraints of 
the lower-level problem (I]), i.e., r u , r d , P, P, af, af and 
af-, at the values estimated in Step 1. Then, recompute the 
parameters defining the utility function, a,b,t and a"j. To this 
end, we make use of the primal-dual reformulation of the 
price-response model 0 ©■ 

Both steps are further described in the subsections below. 
Note that the obtained solution does not guarantee optimality, 
and it is only proved to work satisfactorily in practice (see the 
case study in Section m 


A. Penalty Method 

The so-called penalty method is a convex (linear) relaxation 
of a mathematical programming problem with equilibrium 
constraints, whereby the complementarity conditions of the 
lower-level problem, that is, problem (T), are moved to the 
objective function (4a) of the upper-level problem. Thus, 
we penalize the sum of the dual variables of the inequality 
constraints of problem 0 and their slacks, where the slack 
of a “<”-constraint is defined as the difference between 
its right-hand and left-hand sides, in such a way that the 
slack is always nonnegative. For example, the slack of the 
constraint relative to the maximum pick-up rate ( |Tb[ ) is defined 
as s t = r“ - P t - J^beB x b,t + E-t-i + E&gB x b,t-i, and 
analogously for the rest of the constraints of the lower-level 
problem. 

The penalization can neither ensure that the complemen¬ 
tarity constraints are satisfied, nor that the optimal solution 
of the inverse problem is achieved. Instead, with the penalty 
method, we obtain an approximate solution. In the case study 
of Section IV nonetheless, we show that this solution performs 
notably well. 

After relaxing the complementarity constraints (2d)-(2g). 
the objective function of the estimation problem writes as: 


Minimize E wt{et + e t )+ 
ter 


L ( E wt (^m+^ b,t ■+— E? —~) + 

' beB 
ter 

E «; t (Ar + A? + rr+rf)) 

teT-i ' 


(5a) 
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with the variables being fit = {xt,Ot,ef,e t 


sub j e ct to the following 

constraints: 


’ —i,t i 


(4b) 


g§, © - ©, @ - © 


A“,A?>0 teT-1 

b, t -° bet3,ter. 


(5b) 

(5c) 

(5d) 


The objective function ( |5a| of the relaxed estimation prob¬ 
lem is composed of two terms. The first term represents the 
weighted sum of the absolute values of the deviations of the 
estimated consumption from the measured one. The second 
term, which is multiplied by the penalty term L, is the sum of 
the dual variables of the constraints of the consumers’ price- 
response problem plus their slacks. Note that summing up 
the slacks of the constraints of the consumers’ price-response 
problem eventually boils down to summing up the right-hand 
sides of such constraints. The weights of the estimation errors 
(wt) also multiply the penalization terms. Thus, the model 
weights violations of the complementarity constraints in the 
same way as the estimations errors are weighted. 

Objective function ( |5a| l is subject to the auxiliary constraints 
modeling the absolute value of estimation errors (|4b|>-(|4c|; the 
upper-level-problem constraints imposing monotonically de¬ 
creasing utility blocks <|4d}; and the primal and dual feasibility 
constraints of the lower-level problem, 
and (|5c|-(|5dl>. 

The penalty parameter L should be tuned carefully. We use 
cross-validation to this aim, as described in the case study; we 
refer to Section [TV] for further details. 

Finding the optimal solution to problem ([5]) is computation¬ 
ally cheap, because it is a linear programming problem. On 
the other hand, the optimal solution to this problem might be 
significantly different from the one that we are actually looking 
for, which is the optimal solution to the original estimation 
problem ([4]). Furthermore, the solution to ([5]) depends on the 
user-tuned penalization parameter L, which is given as an 
input and needs to be decided beforehand. 


B. Refining the Utility Function 

In this subsection, we elaborate on the second step of 
the strategy we employ to estimate the parameters of the 
market bid that best captures the price-response of the cluster 
of loads. Recall that this strategy has been briefly outlined 
in the introduction of Section m The ultimate purpose of 
this additional step is to re-estimate or refine the parameters 
characterizing the utility function of the consumers’ price- 
response model 0, namely, «)' and the coefficients a". In 
plain words, we want to improve the estimation of these 
parameters with respect to the values that are directly obtained 
from the relaxed estimation problem <[5}. With this aim in 
mind, we fix the parameters defining the constraints of the 
cluster’s price-response problem ([T} to the values estimated in 
Step 1, that is, to the values obtained by solving the relaxed 
estimation problem (|5j. Therefore, the bounds P t , P t and the 
maximum pick-up and drop-off rates r“, rf are now treated as 
given parameters in this step. Consequently, the only upper- 
level variables that enter the lower-level problem Q, namely. 


the intersects ojj of the various blocks defining the utility 
function and the linear coefficients af, appear in the objective 
function of problem 0- This will allow us to formulate the 
utility-refining problem as a linear programming problem. 

Indeed, consider the primal-dual optimality conditions of 
the consumers’ price-response model that is, the primal 
and dual feasibility constraints and the strong duality condi¬ 
tion. These conditions are also necessary and sufficient for 
optimality due to the linear nature of this model. 

We determine the (possibly approximate) block-wise repre¬ 
sentation of the measured consumption at time t, x ™ eas , which 
we denote by YbeB x b't ar| d is given as a sum of B blocks 

of size Pt J- 1 each. In particular, we define YbeB xr bt as as 
follows: 



^meas 
U X t 


if p t < 


> Pt , teT 


< Pt j ter 


ry.meas 

n 


<p t , ter. 


where each x™ t is determined such that the blocks with higher 
utility are filled first. Now we replace Xt in the primal-dual 


reformulation of 0 with Y 


t . Consequently, the primal 


beB X b,t 

feasibility constraints are ineffective and can be dropped. 

Once Xt has been replaced with YbeB x ™t i n th e primal- 
dual reformulation of (|T]i and the primal feasibility constraints 
have been dropped, we solve an optimization problem (with 
the utility parameters at, and af as decision variables) that 


aims to minimize the weighted duality gap, as in |14|. For 
every time period t in the training data set, we obtain a 
contribution (e t ) to the total duality gap ( YteT £t )’ defined 
as the difference between the dual objective function value 
at time t minus the primal objective function value at time 
t. This allows us to find close-to-optimal solutions for the 
consumers’ price-response model <[T}. Thus, in the case when 
the duality gap is equal to zero, the measured consumption, 
if feasible, would be optimal in <|TJ. In the case when the 
duality gap is greater than zero, the measured consumption 
would not be optimal. Intuitively, we attempt to find values 
for the parameters defining the block-wise utility function such 
that the measured consumption is as optimal as possible for 
problem 0. 

Hence, the utility-refining problem consists in minimizing 
the sum of weighted duality gaps 


Minimize 

_ a b,t,^t A /, teF 




(6a) 


Note that we assign different weights to the duality gaps 
accrued in different time periods, in a way analogous to what 
we do with the absolute value of residuals in 0. Objective 
function ( [6a] ) is subject to 

- Pi X) + £l = Y1 \ Pl (6b) 

beB beB beB 4 A 

~Pt^2 Xb ’t + e ‘ = ( Pl B Pt ) + 

beB beB beB 4 A 

{fit ~Pt + Pt- 1 ) K + {4 + P t ~ Pt- 1 ) 4 t e T-r (6c) 
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(|2a]> — ([2c|) 


(6d) 

& b,t ^ &&+l,£ 

teT 

(6e) 

A?, A? >0 

t G T— i 

(6f) 

V>f > 0 

teT 

(6g) 


The set of constraints ( |6cj ) constitutes the relaxed strong 
duality conditions, which express that the objective function 
of the original problem at time t, previously formulated in 
Equation (jTJ, plus the duality gap at time t, denoted by e t , 
must be equal to the objective function of its dual problem 
also at time t. Equation ( |6b| works similarly, but for t = 1. 
The constraints relative to the dual of the original problem are 
grouped in ( f6d| . Constraint ( [6c] ) requires that the estimated 
utility be monotonically decreasing. Finally, constraints © 
and ([6g| impose the non-negative character of dual variables. 


specifically, outside temperature, solar irradiance, wind speed, 
humidity, dew point and wind direction. Moreover, we include 
0/1 feature variables to indicate the hour of the day, with one 
binary variable per hour (from 0 to 23), and one per day of 
the week (from 0 to 6). A sample of the dataset is shown in 
Figure [T] where the load is plotted in the upper plot, the price 
in the middle plot, and the load versus the outside temperature 
and the dew point in the bottom plots. The lines depicted in 
the bottom plots represent the linear relationship between the 
pairs of variables, and these are negative in both cases. The 
high variability in the price is also noteworthy: from the 1st to 
the 8th of December, the standard deviation of the price is 5.6 
times higher than during the rest of the month ($67.9/MWh 
versus $12.03/MWh). 

A. Benchmark Models 


IV. Case Study 

The proposed methodology to estimate the market bid 
that best captures the price-response of a pool of flexible 
consumers is tested using data from a real-life case study. 
The data relates to the Olympic Peninsula experiment, which 
took place in Washington and Oregon states between May 
2006 and March 2007 [17]. The electricity price was sent 
out every fifteen minutes to 27 households that participated in 
the experiment. The price-sensitive controllers and thermostats 
installed in each house decided when to turn on and off 
the appliances, based on the price and on the house owner’s 
preferences. 
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Fig. 1. The upper and the middle plot show the load and the price, 
respectively. The bottom plots represent the load in the vertical axis versus 
the outside temperature and the dew point, on the left and on the right, 
respectively. The data shown spans from the 4th to the 18th of December. 

For the case study, we use hourly measurements of load 
consumption, broadcast price, and observed weather variables, 
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To test the quality of the market bid estimated by the 
proposed methodology, we quantify and assess the extent 
to which such a bid is able to predict the consumption of 
the cluster of price-responsive loads. For the evaluation, we 
compare two versions of the inverse optimization scheme 
proposed in this paper with the Auto-Regressive model with 
exogenous inputs (ARX) described in j5]|. Note that this time 
series model was also applied by [5j to the same data set of 
the Olympic Peninsula project. All in all, we benchmark three 
different models: 


ARX, which stands for Auto-Regressive model with ex¬ 
ogenous inputs (23) , This is the type of prediction model 
used in 0 and (5). The consumption x f is modeled as a 
linear combination of past values of consumption up to lag 
n, X t - n = {xt, ■ ■ ■, Xt-n}, and other explanatory variables 
Z t = { Z t ,..., Z t -n}. In mathematical terms, an ARX model 
can be expressed as Xt = ‘d x X t - n + + et , with 

et ~ N(0,er 2 ) and er 2 is the variance. 

Simple Inv This benchmark model consists in the utility¬ 
refining problem presented in Section [Tll-B where the param¬ 
eters of maximum pick-up and drop-off rates and consumption 
limits are computed from past observed values of consumption 
in a simple manner: we set the maximum pick-up and drop¬ 
off rates to the maximum values taken on by these parameters 
during the last seven days of observed data. All the features 
are used to explain the variability in the block-wise marginal 
utility function of the pool of price-responsive consumers: 
outside temperature, solar radiation, wind speed, humidity, 
dew point, pressure, and hour and week-day indicators. For 
this model, we use B=12 blocks of utility. This benchmark 
is inspired from the more simplified inverse optimization 
scheme presented in G3 and [ 14] (note, however, that neither 
1151, nor fl4) consider the possibility of leveraging auxiliary 
information, i.e., features, to better explain the data, unlike we 
do for the problem at hand). 

Inv This corresponds to the inverse optimization scheme 
with features that we propose, which runs following the two- 
step estimation procedure described in Section III with B=12 
blocks of utility. Here we only use the outside temperature 
and hourly indicator variables as features. We re-parametrize 
weights wt with respect to a single parameter, called forgetting 
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factor, and denoted as E > 0, in the following manner: 
Wt = gapt (y) for t £ T and T being the total number 
of periods. The variable gap indicates whether the observation 
was correctly measured ( gap = 1) or not ( gap = 0). Parameter 
E indicates how rapidly the weight drops (how rapidly the 
model forgets). When E = 0, the weight of the observations 
is either 1 or 0 depending on the variable gap. As E increases, 
the recent observations weight comparatively more than the 
old ones. 

B. Validation of the Model and Performance in December 

In this subsection we validate the benchmarked models and 
assess their performance during the test month of December 
2006. 

For the sake of simplicity, we assume the price and the 
features to be known for the past and also for the future. 
It is worth noticing, though, that the proposed methodology 
need not a prediction of the electricity price when used for 
bidding in the market and not for predicting the aggregated 
consumption of a cluster of loads. This is so because the 
market bid expresses the desired consumption of the pool of 
loads for any price that clears the market. The same cannot 
be said, however, for prediction models of the type of ARX, 
which would need to be used in combination with extra 
tools, no matter how simple they could be, for predicting the 
electricity price and for optimizing under uncertainty in order 
to generate a market bid. 

There are two parameters that need to be chosen before 
testing the models: the penalty parameter L and the forgetting 
factor E. We seek a combination of parameters such that the 
prediction error is minimized. We achieve this by validating 
the models with past data, in a rolling-horizon manner, and 
with different combinations of the parameters L and E. The 
results are shown in Figure [2] The MAPE is shown on the 
y-axis against the penalty L in the x-axis, with the different 
lines corresponding to different values of the forgetting factor 
E. From this plot, it can be seen that a forgetting factor of 
E = 1 or E = 2 yields a better performance than when there 
is no forgetting factor at all (E = 0), or when this is too high 
(E > 3). We conclude that selecting L = 0.1 and E = 1 
results in the best performance of the model, in terms of the 
MAPE. 



L 


Fig. 2. Results from the validation of the input parameters L and E, to be 
used during December. 

Once the different models have been validated, we proceed 
to test them. For this purpose, we first set the cross-validated 


input parameters to L = 0.1 and E = 1, and then, predict 
the load for the next day of operation in a rolling-horizon 
manner. In order to mimic a real-life usage of these models, 
we estimate the parameters of the bid on every day of the 
test period at 12:00 using historical values from three months 
in the past. Then, as if the market were cleared, we input 
the price of the day-ahead market (13 to 36 hours ahead) in 
the consumers’ price-response model, obtaining a forecast of 
the consumption. Finally, we compare the predicted versus 
the actual realized consumption and move the rolling-horizon 
window to the next day repeating the process for the rest of 
the test period. Similarly, the parameters of the ARX model 
are re-estimated every day at 12:00, and predictions are made 
for 13 to 36 hours ahead. 

Results for a sample of consecutive days, from the 10th 
to the 13th of December, are shown in Figure [3] The actual 
load is displayed in a continuous solid line, while the load 
predictions from the various benchmarked models are shown 
with different types of markers. First, note that the Simple 
Inv model is clearly under-performing compared to the other 
methodologies, in terms of prediction accuracy. Recall that, in 
this model, the maximum and minimum load consumptions, 
together with the maximum pick-up and drop-off rates, are 
estimated from historical values and assumed to remain con¬ 
stant along the day, independently of the external variables (the 
features). This basically leaves the utility alone to model the 
price-response of the pool of houses, which, judging from the 
results, is not enough. The ARX model is able to follow the 
load pattern to a certain extent. Nevertheless, it is not able to 
capture the sudden decreases in the load during the night time 
or during the peak hours in the morning. The proposed model 
(Inv) features a considerably much better performance. It is 
able to follow the consumption pattern with good accuracy. 



Fig. 3. Load forecasts issued by the benchmark models, and actual load, for 
the period between the 10th and the 13th of December. 

The performance of each of the benchmarked models during 
the whole month of December is summarized in Table Q] 
The first column shows the Mean Absolute Error (MAE), 
the second column provides the Root Mean Square Error 
(RMSE), and the third column collects the Mean Absolute 
Percentage Error (MAPE). The three performance metrics 
lead to the same conclusions: that the price-response models 
we propose, i.e., Inv, perform better than the ARX model 
and the Simple Inv model. The results collated in Table [I] 
also yield an interesting conclusion: that the electricity price 
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TABLE I 

Performance measures for the three benchmarked models 
during December. 


s 


MAE RMSE MAPE 
ARX 22.176 27.501 0.2750 



Total load in kW Hour of the day Hour of the day 

Day 

— 2006 - 12-04 - - 2006 - 12-11 


Fig. 4. Averaged estimated block-wise marginal utility function for the Inv 
model (left panel), price in $/kWh (middle panel), and load in kW (right 
panel). The solid lines represent data relative to the 4th of December. Dashed 
lines represent data relative to the 11th of December. 


is not the main driver of the consumption of the pool of 
houses and, therefore, is not explanatory enough to predict 
the latter. We conclude this after seeing the performance 
of the Simp Inv, which is not able to follow the load just 
by modeling the price-consumption relationship by means of 
an utility function. The performance is remarkably enhanced 
when proper estimations of the maximum pick-up and drop-off 
rates and the consumptions bounds are employed. 

The estimated block-wise marginal utility function, aver¬ 
aged for the 24 hours of the day, is shown in the left plot 
of Figure [4] for the Inv model. The solid line corresponds 
to the 4th of December, when the price was relatively high 
(middle plot), as was the aggregated consumption of the pool 
of houses (right plot). The dashed line corresponds to the 11th 
of December and shows that the estimated marginal utility is 
lower, as is the price on that day. 

C. Performance During September and March 

In this section, we summarize the performance of the 
benchmarked models during September 2006 and March 2007. 

In Table |IIJ summary statistics for the predictions are 
provided for September (left side) and March (right side). The 
conclusions remain similar as the ones drawn for the month 
of December. The Inv methodology consistently achieves the 
best performance during these two months as well. 

By means of cross-validation |24) , we find that the user- 
tuned parameters yielding the best performance vary over the 
year. For September, the best combination is L = 0.3, E = 0, 
while for March it is L = 0.3, E = 1. 

The optimized penalization parameter L turns out to be 
higher in September and March than in December. This 
penalization parameter is highly related to the actual flexibility 


TABLE II 

Performance measures for the three benchmarked models 



MAE 

September 

RMSE 

MAPE 

MAE 

March 

RMSE 

MAPE 

ARX 

7.649 

9.829 

0.2350 

17.439 

23.395 

0.2509 

Simple Inv 

14.263 

17.8 

0.4945 

44.687 

54.616 

0.8365 

Inv 

5.719 

8.582 

0.1462 

12.652 

16.776 

0.1952 


featured by the pool of houses. Indeed, for a high enough value 
of the penalty (say L > 0.4 for this case study), violating the 
complementarity conditions associated with the consumers’ 
price-response model o is relatively highly penalized. Hence, 
at the optimum, the slacks of the complementarity constraints 
in the relaxed estimation problem <[5]> will be zero or close 
to zero. When this happens, it holds at the optimum that 
r}‘ = r'j and P t = P t . The resulting model is, there¬ 
fore, equivalent to a linear model of the features, fit by 
least weighted absolute errors. When the best performance 
is obtained for a high value of L, it means that the pool of 
houses does not respond so much to changes in the price. 
On the other hand, as the best value for the penalization 
parameter L decreases towards zero, the pool becomes more 
price-responsive: the maximum pick-up and drop-off rates and 
the consumption limits leave more room for the aggregated 
load to change depending on the price. 

Because the penalization parameter is the lowest during 
December, we conclude that more flexibility is observed 
during this month than during September or March. The reason 
could be that December is the coldest of the months studied, 
with a recorded temperature that is on average 9.4°C lower, 
and it is at times of cold whether when the electric water 
heater is used the most. 

V. Summary and Conclusions 

We consider the market-bidding problem of a pool of 
price-responsive consumers. These consumers are, therefore, 
able to react to the electricity price, e.g., by shifting their 
consumption from high-price hours to lower-price hours. The 
total amount of electricity consumed by the aggregation has to 
be purchased in the electricity market, for which the aggregator 
or the retailer is required to place a bid into such a market. 
Traditionally, this bid would simply be a forecast of the load, 
since the load has commonly behaved inelastic ally. However, 
in this paper, we propose to capture the price-response of 
the pool of flexible loads through a more complex, but still 
quite common market bid that consists of a stepwise marginal 
utility function, maximum load pick-up and drop-off rates, 
and maximum and minimum power consumption, in a manner 
analogous to the energy offers made by power producers. 

We propose an original approach to estimate the parameters 
of the bid based on inverse optimization and bi-level program¬ 
ming. Furthermore, we use auxiliary variables to better explain 
the parameters of the bid. The resulting non-linear problem 
is relaxed to a linear one, the solution of which depends on 
a penalization parameter. This parameter is chosen by cross- 
validation, proving to be adequate from a practical point of 
view. 


















For the case study, we used data from the Olympic Peninsula 
project to asses the performance of the proposed methodology. 
We have shown that the estimated bid successfully models 
the price-response of the pool of houses, in such a way that 
the mean absolute percentage error incurred when using the 
estimated market bid for predicting the consumption of the 
pool of houses is kept in between 14% and 22% for all the 
months of the test period. 

We envision two possible avenues for improving the pro¬ 
posed methodology. The first one is to better exploit the 
information contained in a large dataset by allowing for 
non-linear dependencies between the market-bid parameters 
and the features. This could be achieved, for example, by 
the use of B-splines. The second one has to do with the 
development of efficient solution algorithms capable of solving 
the exact estimation problem within a reasonable amount of 
time, instead of the relaxed one. This could potentially be 
accomplished by decomposition and parallel computation. 
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Appendix 

Next we show how to formulate robust constraints to ensure 
that the estimated minimum consumption be always equal to or 
lower than the estimated maximum consumption. At all times, 
and for all plausible realizations of the external variables, we 
want to make sure that: 

p° + J2 a < p° + a f z ^ t e r, vz iit . (7) 

iex iex 

If (|7]) is not fulfilled, problem (jTJ) is infeasible (and the 
market bid does not make sense). Assuming we know the 
range of possible values of the features, i.e., Z,^ t £ [Z 
(jTJ can be rewritten as: 

P° — P° + Maximize < (cq~ 

Zi '\ _ uex 

s.t. ^<Z( t <Zi 

iex 

Denote the dual variables of the upper and lower bounds 
of Z[ t by <j> it and <f>. respectively. Then, the robust con¬ 
straint © can be equivalently reformulated as: 


p° -p° + J2« t z-^A)< o 

t £ r 

(9a) 

iei 



, , p p 

<Pi,t ~ 9^ =a i ^ a P 
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(9b) 

- 0 

i G X, t G T 

(9c) 


Following the same reasoning, one can obtain the set of 
constraints that guarantees the non-negativity of the lower 
bound and consistent maximum pick-up and drop-off rates. 
We leave the exposition and explanation of these constraints 
out of this paper for brevity. 


- af)Z' t } < 0 , t £ r. 


( 8 ) 


